1. Field of the Invention
The present invention relates to the detection, monitoring, and control of overlay network traffic, including peer-to-peer network traffic.
2. Description of the Related Art
Much of the traffic that flows on, for example, the Internet is encrypted. Much traffic also flows through peer-to-peer (P2P) networks and overlay networks, in which transmission paths that operate as virtual tunnels are established between user nodes that operate as relay points.
The term ‘overlay network’ means a network structured on an underlying network and using the underlying network's data transmission function, but having a routing function differing from that of the underlying network. The term ‘peer-to-peer network’ refers to a network of computers that interact on a peer basis. In a peer-to-peer network of personal computers (PCs), for example, each personal computer performs the same network functions, operating sometimes as a terminal node and sometimes as a relay node. Many wide area peer-to-peer networks are implemented as overlay networks on the Internet.
Peer-to-peer networks and other types of overlay networks may use up a large amount of the underlying network's bandwidth, thereby blocking other users' communication, or may conduct illegal activities such as transmitting illegal information or offering illegal services. It is therefore sometimes necessary to obtain information about the activities of overlay and peer-to-peer networks.
Organizations and persons that need to discover the existence of overlay networks and obtain information about them include law enforcement agencies, for crime deterrence and investigation; government ministries, for supervision of telecommunications business; carriers providing communication services; general users who want to eliminate malicious overlay network software, if it has been installed on their computers without their knowledge; copyright holders and holders of other legal rights in the content transmitted or offered on overlay networks; etc.
The type of information that needs to be obtained about an overlay network includes, for example, the presence of encrypted or tunneled traffic, the communication paths of the traffic, the volume of the traffic, and the communication protocols used, but such information can be difficult to obtain. In fact, it can be difficult to discover even the existence of overlay networks.
One reason for these difficulties is that the length and content of packets may be modified at the nodes of an overlay network, as illustrated in FIG. 1, making it difficult for an Internet service provider (ISP) to assess the packet flow.
In the example shown in FIG. 1, a single packet contains data belonging to different streams and destined to different nodes in the overlay network. An overlay node receiving such packets modifies them by shuffling their contents and transmits the modified packets onward toward the appropriate destinations in the overlay network.
In FIG. 1, packet 101, addressed to personal computer PC50, and packet 102, addressed to personal computer PC60, are transmitted from router R10 to an ISP's router R41. Unknown to the ISP, packet 101 includes data to be transferred from personal computer PC50 to router R20 and personal computer PC60, and packet 102 includes data to be transferred from personal computer PC60 to other destinations (not shown) via routers R20 and R30.
Similarly, packet 201, addressed to personal computer PC50, and packet 202, addressed to personal computer PC60, are transmitted from router R20 to the ISP's router R42, and packet 301, addressed to personal computer PC50, is transmitted from router R30 to the ISP's router R43. These packets include data to be transferred to routers R10 and R30 and personal computers PC50 and PC60. Routers R10, R20, and R30 belong to other ISPs.
When these packets reach personal computers PC50 and PC60, which are nodes on the overlay network, their internal data are recombined according to their next destinations and the recombined data are transmitted back to the ISP as packets 510 to 540 and 610 to 630. Because of the flow separation and integration performed at the nodes PC50, PC60 of the overlay network, by analyzing just the packet traffic it is difficult for the ISP to learn of the existence of the overlay network and determine what type of traffic it is carrying.
To some extent, these problems are addressed by the known art.
Japanese Patent Application Publication (JP) No. 2004-343186 (now Japanese patent No. 3698707) discloses various methods of recognizing peer-to-peer traffic, including: a method that separates peer-to-peer traffic by analysis of its address information (claim 1); a method that involves joining a peer-to-peer service to obtain such address information (claim 2); a function for separating traffic having matching address information (claim 3); improved variations of these methods (claims 4 to 7); and a method that includes analyzing the time stamp, outgoing Internet protocol (IP) address, incoming IP address, outgoing port number, incoming port number and packet size of each packet (claim 8).
In particular, JP 2004-343186 teaches installing a traffic separator to observe traffic in a network as shown in FIG. 2. IP addresses of peer-to-peer nodes are obtained by a decoy terminal connected to the peer-to-peer network and are forwarded to the traffic separator, which manages the peer-to-peer traffic and regular network traffic in separate internal databases (not shown).
One problem with the methods disclosed in JP 2004-343186 is that the information obtained by the traffic separator is limited to information held by nodes that have made contact with the decoy terminal and information about the flow of traffic transmitted and received by such nodes; the traffic separator cannot obtain information about other nodes in the peer-to-peer network. Another problem is that no method of specifically blocking peer-to-peer network traffic without blocking other traffic handled by the peer-to-peer network nodes is provided.
JP 2005-202589 discloses a method in which peer-to-peer node information is obtained by a dummy personal computer to discover peer-to-peer traffic. This method, however, requires knowledge that a peer-to-peer network is deployed on the underlying network and knowledge of the personal computers functioning as peer-to-peer nodes and their IP addresses. The dummy personal computer must also be able to join the peer-to-peer network freely. Accordingly, this method fails to identify peer-to-peer traffic if the personal computers belonging to the peer-to-peer network are unknown, or if authentication is necessary to join the peer-to-peer network.
JP 2005-278176 discloses a network management method in which information related to network connectivity, including samples of user activities, is received and analyzed to construct a parameter-based statistical model that predicts the connectivity between different network areas. Network operation is then simulated under various parameter values, to evaluate network traffic (claim 1). Claim 4 and claims 6 to 10 apply this method to the management of peer-to-peer networks. A problem in this method is the reliability of the statistical model. How to construct an adequately reliable statistical model remains an open question.
JP 2006-506877 discloses a network traffic control method in which peer-to-peer messages are detected by a third party who is free to obtain the content of the peer-to-peer messages. This method is inapplicable when it is difficult to detect peer-to-peer messages, or when their content is encrypted and cannot be decrypted.
JP 2007-019949 discloses a network traffic reduction system that assumes that specific content can be recognized in network traffic. This method is inapplicable when specific content cannot be recognized.
JP 2008-113186 discloses a method for implementing an overlay network to provide network flow with a desired quality of service. This method includes the use of a measurement cost index to control the measurement load involved in obtaining accurate measurements of quality. A precondition for this method is that the network operator can detect and control flow paths, however, so this method is inapplicable when users implement an overlay network on their own.
The present applicant has filed a Japanese patent application (application No. 2007-210866, published as JP 2009-49458) for a system for detecting overlay and peer-to-peer network traffic at a particular node by comparing its inbound and outbound traffic, but there is still an unmet need for an overlay traffic detection system and a traffic monitoring and control system that can discover the existence of an unknown overlay network, and can proceed to identify the overlay network nodes, detect of the volume of overlay network traffic and the communication paths used, and discover the protocol used on the overlay network.